Automatic Differentiation and Backpropagation CS701

This lecture discusses the relationship between automatic differentiation and backpropagation. Automatic differentiation (AD) is a technique that takes an implementation of a numerical function f (computed using floating-point numbers) and creates an implementation of f . We explain several techniques for performing AD. For forward-mode AD, we give an explicit transformation of the program, as well as a way to implement AD using operator overloading. We then reformulate AD as a path problem over a so-called computation graph. The path problem can be solved in either direction: one direction corresponds to forward-mode AD; the other corresponds to reverse-mode AD. Finally, we show how reverse-mode AD forms the basis of the backpropagation algorithm for training a neural net: in backpropagation, the weight adjustments computed for a neural net, a given input, and an expected output are found by performing reverse-mode AD on the computation graph for (i) the neural net, along with (ii) additional computation-graph elements for the sum-of-squared differences of the neural net’s output with the expected output.